Skip to content

fix(skills): name optional Glean tools with a fallback - #51

Draft
steve-calvert-glean wants to merge 2 commits into
mainfrom
skills/tool-names-from-tools-list
Draft

steve-calvert-glean wants to merge 2 commits into
mainfrom
skills/tool-names-from-tools-list

Conversation

@steve-calvert-glean

Copy link
Copy Markdown
Contributor

Problem

A Glean MCP server's tools/list is the only complete list of the tools it serves, and that list differs between deployments. Admins add and remove toolsets, and some deployments serve search as enterprise_search. Our skills hardcoded tool lists anyway: using-glean routed questions to code_search, gmail_search, outlook_search, knowledge_graph_query and others as if every deployment had them, and agents-as-tools.md kept a fixed list of 13 "built-in" tools. A hardcoded list can't stay in sync with a live server.

This also replaces #49, which added a second, shorter Glean skill that triggered on the same questions as using-glean.

Change

  • Skills name search, read_document, chat and employee_search freely. These are on nearly every deployment.
  • Skills name any other Glean tool only with a fallback for when it is missing, such as "code_search if available; otherwise search".
  • using-glean says that tools/list is the source of truth and that the tools it names are examples. Its routing table is split into core tools and "other tools, if available", each with a fallback. The host-specific mcp__glean_<server>__<tool> naming is gone.
  • using-glean gains the rules from feat: add /glean skill for cited company answers at minimal tool cost #49 that weren't already there: run independent calls in parallel, stop once the question is answered, retry a search at most twice, and say when sources are stale or disagree.
  • scripts/check-skill-tool-names.mjs enforces the rule in CI (npm run check:skill-tools, with tests in npm run test:scripts). It found 89 problems in 21 files on main.

How the check decides

Only tool names in backticks count. A non-core tool passes when a sentence that names it states a condition ("if x is available", "when it is in your tool list"), when it sits under a heading that says "if available", when the file's opening text names it with a condition, or when the file is that tool's own reference file. The core set and the list of known tools are constants at the top of the script.

Testing

  • npm test (build, then pluginpack validate for claude, cursor and codex)
  • npm run check:skill-tools
  • npm run test:scripts (14 tests)

Open question

The plugin's relay exposes 8 tools (REMOTE_TOOLS_ALLOWLIST), and code_search, the email tools and the knowledge graph tools are not among them. Through the plugin, those skills now fall back to search. Whether the plugin should expose those tools is a separate decision.

A Glean MCP server's tools/list is the only complete list of its tools, and it differs between deployments: admins add and remove toolsets, and some deployments serve a tool under another name. Skills listed tools such as code_search, meeting_lookup and the email tools as if every deployment had them.

Skills now name search, read_document, chat and employee_search freely, and name any other Glean tool only with a fallback for when it is missing. using-glean says that tools/list is the source of truth, splits its routing table into core tools and tools to use if available, and drops the host-specific mcp__glean tool prefix. It also gains rules to run independent calls in parallel, stop once the question is answered, retry a search at most twice, and say when sources are stale or disagree.
Add scripts/check-skill-tool-names.mjs (npm run check:skill-tools) and run it in CI with its tests (npm run test:scripts). It fails when a skill names a non-core Glean tool without saying the tool may be missing, or uses a host-specific mcp__glean prefix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant